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Abstract: Consider the problem of testing multiple null hypotheses. A clas- 
sical approach to dealing with the multiplicity problem is to restrict attention 
to procedures that control the familywise error rate (FWER), the probabil- 
ity of even one false rejection. However, if s is large, control of the FWER 
is so stringent that the ability of a procedure which controls the FWER to 
detect false null hypotheses is limited. Consequently, it is desirable to consider 
other measures of error control. We will consider methods based on control 
of the false discovery proportion (FDP) denned by the number of false rejec- 
tions divided by the total number of rejections (defined to be if there are 
no rejections). The false discovery rate proposed by Benjamini and Hochberg 
(1995) controls E(FDP). Here, we construct methods such that, for any 7 
and a, P{FDP > 7} < a. Based on p-values of individual tests, we consider 
stepdown procedures that control the FDP, without imposing dependence as- 
sumptions on the joint distribution of the p- values. A greatly improved version 
of a method given in Lehmann and Romano 1 1 ill is derived and generalized to 
provide a means by which any sequence of nondecreasing constants can be 
rescaled to ensure control of the FDP. We also provide a stepdown procedure 
that controls the FDR under a dependence assumption. 

1. Introduction 

In this article, we consider the problem of simultaneously testing a finite number 
of null hypotheses Hi (i = 1, . . . , s). We shall assume that tests based on p- values 
px, . . . ,p s are available for the individual hypotheses and the problem is how to 
combine them into a simultaneous test procedure. 

A classical approach to dealing with the multiplicity problem is to restrict at- 
tention to procedures that control the familywise error rate (FWER) , which is the 
probability of one or more false rejections. In addition to error control, one must 
also consider the ability of a procedure to detect departures from the null hypothe- 
ses when they do occur. When the number of tests s is large, control of the FWER 
is so stringent that individual departures from the hypothesis have little chance 
of being detected. Consequently, alternative measures of error control have been 
considered which control false rejections less severely and therefore provide better 
ability to detect false null hypotheses. 

Hommel and Hoffman ||| and Lehmann and Romano 10] considered the k- 
FWER, the probability of rejecting at least k true null hypotheses. Such an error 
rate with k > 1 is appropriate when one is willing to tolerate one or more false 
rejections, provided the number of false rejections is controlled. They derived single 
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step and stepdown methods that guarantee that the k-FWER is bounded above 
by a. Evidently, taking k — 1 reduces to the usual FWER. Lehmann and Romano 
|in| also considered control of the false discovery proportion (FDP), defined as the 
total number of false rejections divided by the total number of rejections (and equal 
to if there are no rejections). Given a user specified value 7 G (0, 1), control of the 
FDP means we wish to ensure that P{FDP > 7} is bounded above by a. Control 
of the false discovery rate (FDR) demands that E(FDP) is bounded above by a. 
Setting 7 = reduces to the usual FWER. 

Recently, many methods have been proposed which control error rates that are 
less stringent than the FWER. For example, Genovese and Wasserman study 
asymptotic procedures that control the FDP (and the FDR) in the framework 
of a random effects mixture model. These ideas are extended in Perone Pacifico, 
Genovese, Verdinelli and Wasserman 0], where in the context of random fields, 
the number of null hypotheses is uncountable. Korn, Troendle, McShane and Simon 
provide methods that control both the k-FWER and FDP; they provide some 
justification for their methods, but they are limited to a multivariate permutation 
model. Alternative methods of control of the k-FWER and FDP are given in van 
der Laan, Dudoit and Pollard 01 ■ The methods proposed in Lehmann and Romano 
|lf)l | are not asymptotic and hold under either mild or no assumptions, as long as 
p- values are available for testing each individual hypothesis. In this article, we offer 
an improved method that controls the FDP under no dependence assumptions of 
the p-values. The method is seen to be a considerable improvement in that the 
critical values of the new procedure can be increased by typically 50 percent over 
the earlier procedure, while still maintaining control of the FDP. The argument 
used to establish the improvement is then generalized to provide a means by which 
any nondecreasing sequence of constants can be rescaled (by a factor that depends 
on s, 7, and a) so as to ensure control of the FDP. 

It is of interest to compare control of the FDP with control of the FDR, and 
some obvious connections between methods that control the FDP in the sense that 

P{FDP > 7} < a 

and methods that control its expected value, the FDR, can be made. Indeed, for 
any random variable X on [0, 1], we have 

E(X) = E(X\X < j)P{X < 7} + E(X\X > ~{)P{X > 7} 
< lP{X < 7} + p{x > 7} , 

which leads to 

1-7 7 

with the last inequality just Markov's inequality. Applying this to X = FDP, we 
see that, if a method controls the FDR at level q, then it controls the FDP in the 
sense P{FDP > 7} < q/j. Obviously, this is very crude because if q and 7 are 
both small, the ratio can be quite large. The first inequality in says that if 

the FDP is controlled in the sense of then the FDR is controlled at level 

a(l — 7) + 7, which is > a but typically only slightly. Therefore, in principle, a 
method that controls the FDP in the sense of <|3.3|) can be used to control the 
FDR and vice versa. 

The paper is organized as follows. In Section |3 we describe our terminology 
and the general class of stepdown procedures that are examined. Results from 
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Lehmann and Romano are summarized to motivate our choice of critical values. 
Control of the FDP is then considered in Section[3] The main result is presented in 
Theorem 13.41 and generalized in Theorem 13. 51 In Section^ we prove that a certain 
stepdown procedure controls the FDR under a dependence assumption. 



2. A class of stepdown procedures 

A formal description of our setup is as follows. Suppose data X is available from 
some model P £ SI. A general hypothesis H can be viewed as a subset w of Q. For 
testing Hi : P G Ui, i = 1, . . . , s, let I(P) denote the set of true null hypotheses 
when P is the true probability distribution; that is, i £ I(P) if and only if P £ Wj. 

We assume that p- values px, ■ ■ ■ , p s are available for testing Hx,...,H s . Specifi- 
cally, we mean that pi must satisfy 

(2.1) P{Pi < u} < u for any u £ (0, 1) and any P 6 u>i, 

Note that we do not require pi to be uniformly distributed on (0, 1) if Hi is true, 
in order to accomodate discrete situations. 

In general, a p-value pi will satisfy (|2.1(l if it is obtained from a nested set of 
rejection regions. In other words, suppose Si(a) is a rejection region for testing Hi, 
that is, 

(2.2) P{X e Si(a)} < a for all < a < 1, P e uj t 
and 

(2.3) Si (a) C Si (a') whenever a < a . 
Then, the p-value pi defined by 

(2.4) ^ =pi(X)=inf{a: IeS,(a)}. 
satisfies (|2.1|) . 

In this article, we will consider the following class of stepdown procedures. Let 

(2.5) a\ < ai < • • • < a s 

be constants, and let p^ < ■ ■ ■ < p^ denote the ordered p- values. If pm > on, 
reject no null hypotheses. Otherwise, 

(2.6) p (1) < oji, . . . ,p (r) < a r , 

and hypotheses -ff(i), ■ ■ • , #( r ) are rejected, where the largest r satisfying l|2.6|) is 
used. That is, a stepdown procedure starts with the most significant p- value and 
continues rejecting hypotheses as long as their corresponding p-values are small. 

The Holm procedure uses cti = a/(s — i + l) and controls the FWER at level 
a under no assumptions on the joint distribution of the p-values. Lehmann and 
Romano ^(J generalized the Holm procedure to control the k-FWER. Specifically, 
consider the stepdown procedure described in (|2.6() . where we now take 



(2.7) 



!f i<k 



Of course, the oti depend on s and k, but we suppress this dependence in the 
notation. 
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Theorem 2.1 (Hommel and Hoffman [8] and Lehmann and Romano [10]). 

For testing Hi : P 6 uji, i = 1, ...,s, suppose pi satisfies l|2.1ll . The stepdown 
procedure described in Ij2.6|l with cti given by ()2.7|) controls the k-FWER; that is, 

(2.8) Pjreject at least k hypotheses Hi with i £ I(P)} < ol for all P . 

Moreover, one cannot increase even one of the constants cti (for i > k) without 
violating control of the k-FWER. Specifically, for i > k, there exists a joint distri- 
bution of the p-values for which 

(2.9) p {f>(i) ^ "i>-P(2) < a 2 , ■ ■ ■ ,P(i-i) < Oi-i,P(i) < on) = a. 

Remark 2.1. Evidently, one can always reject the hypotheses corresponding to 
the smallest k — 1 p-values without violating control of the k-FWER. However, 
it seems counterintuitive to consider a stepdown procedure whose corresponding 
a.i are not monotone nondecreasing. In addition, automatic rejection of k — 1 hy- 
potheses, regardless of the data, appears at the very least a little too optimistic. To 
ensure monotonicity, our stepdown procedure uses cti = ka/s. Even if we were to 
adopt the more optimistic strategy of always rejecting the hypotheses correspond- 
ing to the k — 1 smallest p-values, we could still only reject k or more hypotheses 
if P(k) < ka/s, which is also true for the specific procedure of Theorem l2.ll 



3. Control of the false discovery proportion 

The number k of false rejections that one is willing to tolerate will often increase 
with the number of hypotheses rejected. So, it might be of interest to control not the 
number of false rejections (or sometimes called false discoveries) but the proportion 
of false discoveries. Specifically, let the false discovery proportion (FDP) be defined 

by 

, . f Numb cr of false rejections j f ^ denominator is > 

/q l\ FDP s lotal number or rejections 

I if there are no rejections 

Thus FDP is the proportion of rejected hypotheses that are rejected erroneously. 
When none of the hypotheses are rejected, both numerator and denominator of 
that proportion are 0; since in particular there are no false rejections, the FDP is 
then defined to be 0. 

Bcnjamini and Hochberg proposed to replace control of the FWER by control 
of the false discovery rate (FDR), defined as 

(3.2) FDR = E(FDP). 

The FDR has gained wide acceptance in both theory and practice, largely be- 
cause Benjamini and Hochberg proposed a simple stepup procedure to control the 
FDR. Unlike control of the k-FWER, however, their procedure is not valid with- 
out assumptions on the dependence structure of the p-values. Their original paper 
assumed the very strong assumption of independence of p-values, but this has been 
weakened to include certain types of dependence; see Benjamini and Yekutieli 
In any case, control of the FDR does not prohibit the FDP from varying, even if 
its average value is bounded. Instead, we consider an alternative measure of control 
that guarantees the FDP is bounded, at least with prescribed probability. That is, 
for a given 7 and a in (0, 1), we require 

(3.3) P{FDP > 7} < a. 



On the false discovery proportion 



37 



To develop a stepdown procedure satisfying i|3.3|) . let / denote the number of 
false rejections. At step i, having rejected i — 1 hypotheses, we want to guarantee 
//* < 7, i- e - / < l_7*J; where [x\ is the greatest integer < x. So, if k = [ji\ + 1, 
then f > k should have probability no greater than a; that is, we must control the 
number of false rejections to be < k. Therefore, we use the stepdown constant on 
with this choice of k (which now depends on i); that is, 

(3.4) a t = 



s + [ji\ + l — i 

Lehmann and Romano 1 1 Ol ] give two results that show the stepdown procedure 
with this choice of oti satisfies l|3.3|) . Unfortunately, some joint dependence as- 
sumption on the p- values is required. As before, pi,...,p s denotes the p- values 
of the individual tests. Also, let <?i, ... ,9171 denote the p-values corresponding to 
the |/| = \I(P)\ true null hypotheses. So qi — pj i7 where ji, . . . ,jin correspond to 
the indices of the true null hypotheses. Also, let fx, ... , ?s-\i\ denote the p- values 
of the false null hypotheses. Consider the following condition: for any i = 1, . . . , \I\, 

(3.5) P{q~i < u|fi,...,f s _| 7 |} < it; 

that is, conditional on the observed p- values of the false null hypotheses, a p-value 
corresponding to a true null hypothesis is (conditionally) dominated by the uniform 
distribution, as it is unconditionally in the sense of (|2.1I) . No assumption is made 
regarding the unconditional (or conditional) dependence structure of the true p- 
values, nor is there made any explicit assumption regarding the joint structure of 
the p- values corresponding to false hypotheses, other than the basic assumption 
(I3.5|l . So, for example, if the p- values corresponding to true null hypotheses are 
independent of the false ones, but have arbitrary joint dependence within the group 
of true null hypotheses, the above assumption holds. 

Theorem 3.1 (Lehmann and Romano [10]). Assume the condition l|3.5|l . 
Then, the stepdown procedure with a<i given by (|3.4|l controls the FDP in the sense 
of (TPI) . 

Lehmann and Romano ^(| a ^ so show the same stepdown procedure controls 
the FDP in the sense of (|3.3Jl under an alternative assumption involving the joint 
distribution of the p-values corresponding to true null hypotheses. We follow their 
approach here. 

Theorem 3.2 (Lehmann and Romano |lol|L Consider testing s null hypothe- 
ses, with \I\ of them true. Let $m < • • • < denote the ordered p-values for the 
true hypotheses. Set M — min([7sj + 1, \I\). 

(i) For the stepdown procedure with at given by H3.4(l . 

M 

(3-6) P{FDP> 7 }<P{\J{q {i) < 1 ^-}}. 

i=i ' ' 

(ii) Therefore, if the joint distribution of the p-values of the true null hypotheses 
satisfy Simes inequality; that is, 

P «<7(D < ^} LKfo) ^ ^> U • • • IK*(M) ^ a » ^ a ' 

then P{FDP > 7} < a. 
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Simcs inequality is known to hold for many joint distributions of positively de- 
pendent variables. For example, Sarkar and Chang [151 ] and Sarkar [13J have shown 
that the Simes inequality holds for the family of distributions which is characterized 
by the multivariate positive of order two condition, as well as some other important 
distributions. 

However, we will argue that the stepdown procedure with cti given by (|3.4[1 does 
not control the FDP in general. First, we need to recall Lemma 3.1 of Lehmann 
and Romano [l(ij ]. stated next for convenience (since we use it later as well). It is 
related to Lemma 2.1 of Sarkar [lii ]. 

Lemma 3.1. Suppose pi, . . . ,p t are p-values in the sense that P{pi < u} < u for 
all i and u in (0, 1). Let their ordered values be p^ < • •• < P(t)- Let = (3q < Pi < 
fa < ■ ■ ■ < Pm < 1 for some m <t. 

(i) Then, 

m 

(3.7) P{{p (1) < /?l}|J{P(2) < fo} (J • • ■ \J{P(m) < Pm}} < " 

i=l 

(ii) As long as the right side of (|3.7() is < 1, the bound is sharp in the sense 
that there exists a joint distribution for the p-values for which the inequality is an 
equality. 

The following calculation illustrates the fact that the stepdown procedure with 
cti given by l|3.4|l does not control the FDP in general. 

Example 3.1. Suppose s = 100, 7 = 0.1 and |/| = 90. Construct a joint distri- 
bution of p-values as follows. Let q^ < ■ ■ ■ < g(g ) denote the ordered p-values 
corresponding to the true null hypotheses. Suppose these 90 p-values have some 
joint distribution (specified below). Then, we construct the p-values corresponding 
to the 10 false null hypotheses conditional on the 90 p-values. First, let 8 of the 
p-values corresponding to false null hypotheses be identically zero (or at least less 
than a/100). If q^ < a/92, let the 2 remaining p-values corresponding to false 
null hypotheses be identically 1; otherwise, if q^ > a/92, let the 2 remaining p- 
values also be equal to zero. For this construction, FDP > 7 if gm < a/92 or 
9(2) < 2a/91. The value of 

a I I 2a 

can be bounded by Lemma 3.1. The lemma bounds this expression by 

9o(— + itdk) ~ i. 48q > a , 
\92 2 J 

Moreover, Lemma 3.1 gives a joint distribution for the 90 p-values corresponding 
to true null hypotheses for which this calculation is an equality. 

Since one may not wish to assume any dependence conditions on the p-values, 
Lehmann and Romano |l0| use Theorem 13.21 to derive a method that controls the 
FDP without any dependence assumptions. One simply needs to bound the right 
hand side of l|3.6|l . In fact, Hommel |7j has shown that 

n\J{m< l ^}}<aY,T 

i=l 1 1 i=l 
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This suggests we replace a by a(X)i=i(V*)) But of course |/| is unknown. So 
one possibility is to bound |/| by s which then results in replacing a by a/C s , where 



(3.8) Cj = 



Clearly, changing a in this way is much too conservative and results in a much less 
powerful method. However, notice in Ij3.6|l that we really only need to bound the 
union over M < [ 7 s] + 1 events. This leads to the following result. 

Theorem 3.3 (Lehmann and Romano [10]). For testing Hi : P G uj it 
i = 1, . . . , s, suppose pi satisfies <|2.1[l . Consider the stepdown procedure with con- 
stants a'i = ai/Cy ls \ i+ i, where cti is given by (|3.4|l and Cj defined by (|3.8|l . Then, 
P{FDP > 7} < a. 

The next goal is to improve upon Theorem 13.31 In the definition of a[ , a, is 
divided by C|-ysJ+i • Instead, we will construct a stepdown procedure with constants 
= cti/D, where D = _D(7,a, s) is much smaller than Ci 7S j +1 . This procedure 
will also control the FDP but, since the critical values a" are uniformly bigger than 
the a'i, the new procedure can reject more hypotheses and hence is more powerful. 

To this end, define 

Tfl 

(3.9) (3 m = — m = 1, . . . , [ 7 sJ 

maxjs + m — \ ^ \ + 1, 



LtsJ + 1 



and 

(3.10) /3 L7sJ + 

where \x~\ is the least integer > x. 
Next, let 

(3.11) N = JV( 7 , a, \I\) = min{L 7S J + 1, |7|, L 7 (^-^ + 1)J + 1}. 

1-7 

Then, let (3q = and set 

N R - Ft 

(3.12) 5 = 5(7, S ,|/|) = |/| V ^ " ""' 



=1 



Finally, let 

(3.13) D = D(j,s) =maxS'( 7 ,s, |7|). 

Theorem 3.4. For testing Hi : P G ^i, i = l,...,s, suppose pi satisfies l|2.1|l . 
Consider the stepdown procedure with constants a" — ai/D(-y, s), where ai is given 
by and L>( 7 , s) is defined by (J2H3J). Then, P{FDP > 7} < a. 

Proof. Let a" = a/D. Denote by 

9(1) < •■• <9(|/|) 



the ordered p- values corresponding only to true null hypotheses. Let j be the small- 
est (random) index where the FDP exceeds 7 for the first time at step j; that is, the 
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number of false rejections out of the first j —I rejections divided by j exceeds 7 for 
the first time at j. Denote by m > the unique integer satisfying m— 1 < <yj < m. 
Then, at step j, it must be the case that m true null hypotheses have been rejected. 
Hence, 

„ ma" 

qtm) < OLj = — ; 

y ' J s + m — j 
Note that the number of true hypotheses |/| satisfies 

|/| < s + m — j. 

Further note that jj < m implies that 

(3-14) j < [-1 - 1. 

7 

Hence, a'- is bounded above by f3 m defined by l|3.9|) whenever m — 1 < jj < m. 
Note that, when m = [t s J + 1, we bound a" by using j < s rather than (|3.14() . 

The possible values of m that must be considered can be bounded. First of all, 
j < s implies that m < [7s] + 1. Likewise, it must be the case that m < |/|. Finally, 
note that j > s 1 ^ 7 ^ implies that FDP > 7. To see this, observe that 

1^ = (- + r^(- 

so at such a step j, it must be the case that 

t>-J-(a-\I\) 
1-7 

true null hypotheses have been rejected. If we denote by / = j — t the number of 
false null hypotheses that have been rejected at step j, it follows that 

t>rr ! —f, 

1-7 

which in turn implies that 

FDP =ih >r 

Hence, for j to satisfy the above assumption of minimality, it must be the case that 

1-7 

from which it follows that we must also have 



m < hC 1 ^ + i)J + 1. 
1-7 

Therefore, with N defined in 1 If) and j defined as above, we have that 

N 

P{FDP >j}<J2 P ^ a i> fK m ~ 1 ^ 13 < m >} 

m— 1 

N 

<^P {«(m) < a/' 0m} f|{m - 1 < 7i < m}} 
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N ( N ~\ 

m=l U=l J 

p |u%) <«"&}• 



< 



Note that /3 m < f3 m +i- To see this, observed that the expression to + s— \—\ + 1 
is monotone nonincreasing in to, and so the denominator of (3 mi max{m + s— \— ] + 
1, |/|}, is monotone nonincreasing in to as well. Also observe that (3 m < rn/\I\ < 1 
whenever m < N. We can therefore apply Lemma 13. II to conclude that 



P{FDP > 7} < a"\I\^2 



i=l 

where 5" and D are defined in 1|3.12[) and (|3. 13|> . respectively. □ 

It is important to note that by construction the quantity D(-y, s), which is defined 
to be the maximum over the possible values of |7| of the quantity 5(7, s, |/|), does 
not depend on the unknown number of true hypotheses. Indeed, if the number of 
true hypotheses, |/|, were known, then the smaller quantity 5(7, s, |/|) could be 
used in place of D(j, s). 

Unfortunately, a convenient formula is not available for £>(7, s), though it is 
simple to program its evaluation. For example, if s = 100 and 7 = 0.1, then 
D = 2.0385. In contrast, the constant Ci 7s j +1 = C\\ = 3.0199. In this case, the 
value of |/| that maximizes S to yield D is 55. Below, in Table ^ we evaluate 
D(-f,s) and C|_ 7S j + i for several different values of 7 and s. We also compute the 
ratio of C^^j+i to -D(7, s), from which it is possible to see the magnitude of the 
improvement of the Theorem 13.41 over Theorem 13.31 the constants of Theorem 13.41 
are generally about 50 percent larger than those of Theorem 13.31 

Remark 3.1. The following crude argument suggests that, for critical values of the 
form don for some constant d, the value of d = D _1 (7, s) is very nearly the largest 
possible constant one can use and still maintan control of the FDP. Consider the 
case where s — 1000 and 7 = .1. In this instance, the value of |/| that maximizes S 
is 712, yielding N = 33 and D = 3.4179. Suppose that |/| = 712 and construct the 
joint distribution of the 288 p- values corresponding to false hypotheses as follows: 
For 1 < i < 28, if < afli and > a(3j for all j < i, then let [~^] — 1 of the 
false p- values be and set the remainder equal to 1. Let the joint distribution of 
the 712 true p- values be constructed according to the configuration in Lemma 13. II 
Note that for such a joint distribution of p- values, we have that 

P{FDP > 7} > P j |J {q t < a/3i}\ = a\I\ £ = 3.2212a. 

U=l J i=l 

Hence, the largest one could possibly increase the constants by a multiple and still 
maintain control of the FDP is by a factor of 3.4179/3.2212 w 1.061. 
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Table 1 
Values 0/0(7, s) an d C17SJ+1 



s 


7 


D( 7 ,s) 


C\ -y S 1 +1 
1 - 


Ratio 


100 


0.01 


1 


1.5 


1.5 


250 


0.01 


1.4981 


1.8333 


1.2238 


500 


0.01 


1.7246 


2.45 


1.4206 


1000 


0.01 


2.0022 


3.0199 


1.5083 


2000 


0.01 


2.3515 


3.6454 


1.5503 


5000 


0.01 


2.8929 


4.5188 


1.562 


25 


0.05 


1.4286 


1.5 


1.05 


50 


0.05 


1.4952 


1.8333 


1.2262 


100 


0.05 


1.734 


2.45 


1.4129 


250 


0.05 


2.1237 


3.1801 


1.4974 


500 


0.05 


2.4954 


3.8544 


1.5446 


1000 


0.05 


2.9177 


4.5188 


1.5488 


2000 


0.05 


3.3817 


5.1973 


1.5369 


5000 


0.05 


4.0441 


6.1047 


1.5095 


10 


0.1 


1 


1.5 


1.5 


25 


0.1 


1.4975 


1.8333 


1.2242 


50 


0.1 


1.7457 


2.45 


1.4034 


100 


0.1 


2.0385 


3.0199 


1.4814 


250 


0.1 


2.5225 


3.8544 


1.528 


500 


0.1 


2.9502 


4.5188 


1.5317 


1000 


0.1 


3.4179 


5.1973 


1.5206 


2000 


0.1 


3.9175 


5.883 


1.5017 


5000 


0.1 


4.6154 


6.7948 


1.4722 



It is worthwhile to note that the argument used in the proof of Theorem 13.41 
does not depend on the specific form of the original a.i. In fact, it can be used with 
any nondecreasing sequence of constants to construct a stepdown procedure that 
controls the FDP by scaling the constants appropriately. To see that this is the 
case, consider any nondecreasing sequence of constants 5i < • • • < 6 S such that 
< Si < 1 (this restriction is without loss of generality since it can always be 
acheived by rescaling the constants if necessary) and redefine the constants [3 m of 
equations (|3.9I) and (|3.1U|) by the rule 

(3.15) f3 m = <5fe( s , 7 ,m,|/|) m = 1, . . . , [_7s| + 1 

where 

Tfi 

k(s, 7, m, |/|) = min{s, s + m — |/|, [ — ] — 1}. 

7 

Note that in the special case where Si = cti, the definition of (5 m in equation l|3.15|l 
agrees with the earlier definition of equations (|3.9II and (|3.1Uf) . Maintaining the 
definitions of N, S, and D in equations l|3.11() - l|3.13|l (where they are now defined 
in terms of the (3 m sequence given by equation l|3.15|l ). we then have the following 
result: 

Theorem 3.5. For testing Hi : P G u>i, i = 1, . . . , s, suppose pi satisfies (|2.1|) . Let 

5i < ■ ■ ■ < 5 S be any nondecreasing sequence of constants such that < Si < 1 and 
consider the stepdown procedure with constants 5" = aSi/D(j, s), where -D(7, s) is 
defined by (|3~T3|) . Then, P{FDP > 7} < a. 

Proof. Define j and m as in the proof of Theorem 13.41 We have, as before, that 
whenever m — 1 < < m 

I J| < s + m - j, 

and 

rTU-, 

j^r-i-i- 
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Since j < s, it follows that 

<?(m) < Sj < (3 m , 

where f3 m is as defined in (|3.15|) . The remainder of the argument is identical to the 
proof of Theorem 13.41 so we do not repeat it here. □ 

As an illustration of this more general result, consider the nondecreasing se- 
quence of constants given simply by r/i — -. These constants are proportional to 
the constants used in the procedures for controlling the FDR by Benjamini and 
Hochberg 0] and Benjamini and Yekutieli Q. Applying Theorem 13.51 to this se- 
quence of constants yields the following corollary: 

Corollary 3.1. For testing Hi : P G u>i, i = l,...,s, suppose pi satisfies 12.1fl . 
Then the following are true: 

(i) The stepdown procedure with constants r/l = arji/ D("f, s), where -D(7, s) is defined 
by (|3~T3|) . satisfies P{FDP > 7} < a; 

(ii) The stepdown procedure with constants n" = jarji/ max{C^ 7S j, 1}, where Cq is 
understood to equal 0, satisfies P{FDP > 7} < a. 

Proof. The proof of (i) follows immediately from Theorem 13.51 To prove (ii), first 
observe that N < [-fs\ + 1 and that for this particular sequence, we have that 
Pm < nfinj^j, 1} = : Cm- Hence, we have that 

N 1/ysJ+i 

P{\J{Q(m) <M}<P{ (J ^(m)< Cm}}- 
i— 1 m— 1 

Using Lemma f3. II we can bound the righthand side of this inequality by the sum 

L7SJ+1 _ t 

m— 1 

Whenever \_"fs\ > 1, we have that C[ 7 sj+i = C[7sj = s, so this sum can in turn be 
bounded by 

" — " ^ — — — C|ts|- 
7s L — ' m 7 

m— 1 

If, on the other hand, [7-sJ = 0, we can simply bound the sum by — . Therefore, if 
we let Co = 0, we have that 

D(j,s) < imax{C L7sJ ,l}, 

from which the desired claim follows. □ 

In summary, given any nondecreasing sequence of constants Si, we have derived 
a stepdown procedure which controls the FDP, and so it is interesting to compare 
such FZ)P-controlling procedures. Clearly, a procedure with larger critical values is 
preferable to one with smaller ones, subject to the error constraint. The discussion 
from Remark 3.1 leads us to believe that the critical values from a single procedure 
will not uniformly dominate those from another, at least approximately. We now 
consider some specific comparisons which may shed light on how to choose among 
the various procedures. 
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Table 2 

Values of D(-y, s) and — max{C^ 7S j , 1} 



s 


7 


D( 7 ,s) 


imax{C L7sJ ,l} 
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«9 nfizL 
uz.uu^ 


2 1714 


^nn 
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u.uo 


O ( . Jlo 


7fi 31 Q 




1000 


0.05 


47.26 


89.984 


1.904 


2000 


0.05 


57.666 


103.75 


1.7991 


5000 


0.05 


72.126 


122.01 


1.6917 


10 


0.1 


3 


10 


3.3333 


25 


0.1 


6.4 


15 


2.3438 


50 


0.1 


9.3867 


22.833 


2.4325 


100 


0.1 


13.02 


29.29 


2.2496 


250 


0.1 


18.834 


38.16 


2.0261 


500 


0.1 


23.703 


44.992 


1.8981 


1000 


0.1 


28.886 


51.874 


1.7958 


2000 


0.1 


34.317 


58.78 


1.7129 


5000 


0.1 


41.775 


67.928 


1.6261 



To compare the constants from parts (i) and (ii) of Corollary 13.11 Table [21 dis- 
plays -D(7, s) and — max{C[ 7S j , 1} for several different values of s and 7, as well as 
the ratio - max{C|^ 7S j , 1}/D(7, s). In this instance, the improvement between the 
constants from part (i) and part (ii) is dramatic: The constants r}[ are often at least 
twice as large as the constants 77". 

It is also of interest to compare the constants from part (i) of the corollary with 
those from Theorem 13.41 We do this for the case in which s = 100, 7 = .1, and 
a = .05 in Figure^ The top panel displays the constants a" from Theorem l3.4l and 
the middle panel displays the constants 77^ from Corollarv l3.ll (i) . Note that the scale 
of the top panel is much larger than the scale of the middle panel. It is therefore 
clear that the constants a" are generally much larger than the constants 7][ . But it 
is important to note that the constants from Theorem 13.41 are not uniformly larger 
than the constants from Corollary 13. II (i). To make this clear, the bottom panel of 
Figure ^ displays the ratio a"/^. Notice that at steps 7 - 9, 15 - 19, and 25 - 29 
the ratios are strictly less than 1, meaning that at those steps the 77^ are larger than 
the a". Following our discussion in Remark l3 . 1 1 that these constants are very nearly 
the best possible up to a scalar multiple, we should expect that this would be the 
case because otherwise the constants r)[ could be multiplied by a factor larger than 
1 and still retain control of the FDP. Even at these steps, however, the constants 
ril are very close to the constants a" in absolute terms. Since the constants a" 
are considerably larger than the constants 77^ at other steps, this suggests that the 
procedure based upon the constants a" is preferrable to the procedure based on 
the constants 77^. 

4. Control of the FDR 

Next, we construct a stcpdown procedure that controls the FDR under the same 
conditions as Theorem 13.11 The dependence condition used is much weaker than 
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10 20 30 40 50 60 70 80 90 100 



Fig 1. Stepdown Constants for s = 100, 7 = .1, and a = .05. 



that of independence of p- values used by Benjamini and Liu |2|. 

Theorem 4.1. For testing Hi : P G a;,, i = l,...,s, suppose pi satisfies (|2.1|l . 
Consider the stepdown procedure with constants 

(4.1) a* = min{ (s _^_ 1)2 , 1} 

and assume the condition l|3.5|) . Then, FDR < a. 

Proof. First note that if |/| = 0, then FDi? = 0. Second, if \I\ = s, then FDR = 
P{P(D <a{}< £- =1 Pfe < a\} < sal = a. 

Now suppose that < |7| < s. Define q\, . . . , q\i\ and fi, . . . , r s _|_r| to be the p- 
values corresponding, respectively, to the true and false hypotheses, and let < 
■ • • < an< i ^(1) — ' ' ' — ?{s-\i\) be their ordered values. Denote by j the largest 
index such that < a*, . . . , fy) < a* (defined to be if > a*). Define i to 
be the total number of true hypotheses rejected by the stepdown procedure and 
/ to be the total number of false hypotheses rejected by the stepdown procedure. 
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Using this notation, observe that 



E(FDP\h, r s -\i\) = ^(^{* + f> ■ ■ ■ .^-m) 

<E(-^{t>Q}\h,...,f.-\i\) 
t + j 

<T^E({t>0}\h,...,r s -\i\) 

1 i ^ t=i 



( 4 - 2 ) < 17^41^1^+1 



\i\+r 



\I\ 2 . r sa 



(4.3) < 



(s-j)(\I\+j)(s-jY 



The inequality 1)4. 2|) follows from the assumption l|3.5[l on the joint distribution 
of p-values. To complete the proof, note that \I\ + j < s. It follows that (jrrjy < a 
and (|7| + j)(s — j) — \I\s = j(s — \I\) — j 2 = j(s — \I\ — j) > 0. Combining these 
two inequalities, we have that the expression in (|4.3|) is bounded above by a. The 
desired bound for the FDR follows immediately □ 

The following simple example illustrates the fact that the FDR is not controlled 
by the stepdown procedure with constants a* absent the restriction l|3.5[l on the 
dependence structure of the p- values. 

Example 4.1. Suppose there are s = 3 hypotheses, two of which are true. In this 
case, a\ — ^, a* 2 = and a\ = min{3a, 1}. Define the joint distribution of the 
two true p- values qi and <?2 as follows: Denote by h the half open interval \ s ^, §) 
and let (q±, q?) ~ U(IiX Ij) with probability | for all (i, j) such that «^j,l<«<3 
and 1 < j < 3. It is easy to see that (<?(i), 9(2)) ~ U(Ii x 7j) with probability | for 
all such that i < j, 1 < i < 3 and 1 < j < 3. Now define the distribution 

of the false p-value r% conditional on (91,(72) by the following rule: If qm < a/3, 
then let n = 1; otherwise, let ri = 0. For such a joint distribution of (91, 32, we 
have that the FDP is identically one whenever 9m < § and is at least \ whenever 

f < 9(i) < X' Hence > 

„„„ „ ^ a , 1 „ , a 3a , 

FDE > P{q (1) <-} + -P{- < <? (1) < — }. 

For a < |, we therefore have that 

2a ,3a a, 13a 

FDR ^T + i T-3^l2 >a - 
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Remark 4.1. Some may find it unpalatable to allow the constants to exceed a. 
In this case, one might consider replacing the constants a* above with the more 
conservative values amin{ ( s _* +1 yj , 1}, which by construction are always less than 
a. Since these constants are uniformly smaller than the a*, our method of proof 
shows that the FDR would still be controlled under the dependence condition l|3.5|) . 
The above counterexample, which did not depend on the particular value of aj, 
however, would show that it is not controlled in general. 

Under the dependence condition 1|3.5|) . the constants l|4.1J) control the FDR in 
the sense FDR < a, while the constants given by (|3.4|) control the FDP in the 
sense of (|3 . 31) . Utilizing we can use the constants 14.1fl to control the FDP 

by controlling the FDR at level aj. In Figure |3 we plot the constants 13. 4|) and 
(|4.ip for the special case in which ,s — 100 and we use both constants to control the 
FDP for 7 = .1, and a = .05. 

The top panel displays the constants cm , the middle panel displays the constants 
a*, and the bottom panel displays the ratio onja*. Since the ratios essentially 
always exceed 1, it is clear that in this instance the constants (|3.4|l are superior to 

a 

0.05 | ! 1 ! 1 ! 1 ! ! ! 

0.04 - 

0.03 - ..' - 

0.02 - 

o.o" - ......... 

10 20 30 40 50 60 70 80 90 100 

i 

a. 

0.6 1 1 1 1 1 1 1 1 1 1 1 

0.4 - - 
0.2 - 

Q I 1 1 1 1 1 j 1 ] 1. .■••** 

10 20 30 40 50 60 70 80 90 100 

i 

a/a. 

30 1 1 1 1 r 1 1 1 1 1 1 

20 - 

10 



0' ' ' ' ' ' ' ' ' ' 

10 20 30 40 50 60 70 80 90 100 

i 

Fig 2. FDP Control for s = 100, 7 = .1, and a = .05. 
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a 

0.03 | ! ! 1 1 ! ! 1 r- 

0.02 - 
0.01 ■ 

o' * ' ' ' ' ' 
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i 

a 

1 | 1 1 , , 1 1 , r- 

0.8 

0.6 - ^ 
0.4 - 
0.2 - 

0' ■ ■ ■ ■ ■ ' 1 
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i 

a./a. 

1 | 1 1 1 1 1 1 1 r- 

0.8 

0.6 - 

0.4 

0.2 ' 

I 1 1 , , i i , L_ 

10 20 30 40 50 60 70 80 

i 

Fig 3. FDR Control for s = 100 and a = .05. 

the constants (|4.1fl . If by utilizing we use the constants (|3.4fl to control the 
FDR, on the other hand, we find that the reverse is true. Control of the FDR 
at level a can be achieved, for example, by controlling the FDP at level and 
letting 7 = §■ Figure |31 plots the constants l|3.4|) and l|4.1|l for the special case in 
which s — 100 and we use both constants to control the FDR at level a = .05. 

As before, the top panel displays the constants «i, the middle panel displays the 
constants a*, and the bottom panel displays the ratio ai/a*. In this case, the ratio 
is always less than 1. Thus, in this instance, the constants a* are preferred to the 
constants o^. Of course, the argument used to establish l|l.lfl is rather crude, but 
it nevertheless suggests that it is worthwhile to consider the type of control desired 
when choosing critical values. 

5. Conclusions 

In this article we have described stepdown procedures for testing multiple hypothe- 
ses that control the FDP without any restrictions on the joint distribution of the 
p-values. First, we have improved upon a method proposed by Lehmann and Ro- 
mano 10]. The new procedure is a considerable improvement in the sense that its 
critical values are generally 50 percent larger than those of the earlier procedure. 
Second, we have generalized the method of argument used in establishing this im- 
provement to provide a means by which any nondecresing sequence of constants 
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90 100 
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can be rescaled so as to ensure control of the FDP. Finally, we have also described 
a procedure that controls the FDR, but only under an assumption on the joint 
distribution of the p- values. 

In this article, we focused on the class of stepdown procedures. The alternative 
class of stepup procedures can be described as follows. Let 

(5.1) Qi < a 2 < ■ ■ ■ < a s 

be a nondecreasing sequence of constants. If p^ < a s , then reject all null hy- 
potheses; otherwise, reject hypotheses -ff(i), ■ ■ ■ , -ff( r ) where r is the smallest index 
satisfying 

(5.2) p (s) > a s , ...,P( r+ i) > Or +1 . 

If, for all r, p^ > a r , then reject no hypotheses. That is, a stepup procedure 
begins with the least significant p- value and continues accepting hypotheses as long 
as their corresponding p- values are large. If both a stepdown procedure and stepup 
procedure are based on the same set of constants ai, it is clear that the stepup 
procedure will reject at least as many hypotheses. 

For example, the well-known stepup procedure based on on = ia/s controls the 
FDR at level a, as shown by Benjamini and Hochberg under the assumption 
that the p-values are mutually independent. Benjamini and Yekutieli |3j generalize 
their result to allow for certain types of dependence; also see Sarkar |l4| . Benjamini 
and Yekutieli |3| also derive a procedure controlling the FDR under no dependence 
assumptions. Romano and Shaikh |l2j| derive stepup procedures which control the 
k-FWER and the FDP under no dependence assumptions, and some comparisons 
with stepdown procedures are made as well. 
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